Web search is becoming an increasingly important tool in the lives of millions of individual users to identify relevant content items from among the billions of content items that are accessible on the Internet. These content items are highly diversified in terms of content, format and quality and ranking content items from the web to find the most relevant content item in response to a given query from such a huge amount of diversified data is one of the basic challenges for search engines. Accordingly, machine learning of relevance functions has become a widespread solution to ranking content items where labeled training data is available. Labeled data comprises one or more query-content item pairs with an accompanying judgment as to the relevance of the pair.
When labeling web search results, a given user (who may be a relevance expert) provides a label in accordance with certain enumerated guidelines. These guidelines roughly measure an observed relevance between a query and a content item, e.g., a web page. Moreover, such labeling is usually performed by experts in assessing the relevance of a content item to a query. Such expert-judged training examples have a number of advantages. For example, the queries for judgment may be carefully selected for different purposed and the judged results can be thoroughly verified and tuned. Due to such properties, expert-judged training examples are often regarded as the “gold set” for use in training of a machine learned relevance function. It is very costly, however, for humans to manually label a large number of web search examples, which is required for effectively learning complicated models such as those for modeling web-based content item relevance. There are those that may also argue that the opinions of a few experts regarding relevance may not be representative enough for a highly diverse user population.
Recently, implicit relevance judgments have been derived from user clickthrough logs, which is an appealing source of training data for machine learning of content item relevance functions due to the low cost of obtaining such relevance examples. Expert-judged relevance and clickthrough data, however, have significant difference due to the data sources and the methods of gathering the data. Specifically, compared to expert-judged examples, user preference data (clickthrough data) can be more dynamic and therefore better represent the “epidemic” topics such as movies, fashion, recent events, user group biased searches, etc.
One unique feature of user preference data is so called “reigonality”—for a given query users from a given region may be more interested in a particular URL than users from other regions. This may result in a significant difference between user preference and expert judgment. Some distinct properties of user preference data versus expert judgment are summarized as follows:                Expert judged relevance samples are often small scale and costly to obtain, reflecting the relevance judgment of a small number of people who follow non-volatile judgment criteria. These relevance judgments are usually also labeled in accordance with an absolute ranking.        Implicit relevance samples (clickthrough data or user preference data) are inexpensive to obtain in large volumes and reflect the opinions of a large number of users over the search results they receive from a search engine. One drawback is that this data is often noisy and may possibly change as a function of time, but nonetheless captures most users' interest. Additionally, using such data only provides preferences in the form of a pair of content items (d1, d2), where content item d1 is more relevant to the query q than d2.        
Due to the differing nature of these two types of data, it is possible that they are inconsistent modeling content item relevance functions. Thus, systems, methods and computer program products are needed to take advantage of and combine both expert-judged relevance samples and user preference data when training a model to determine a content item relevance function.